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OVISRVIEW 

In 1963, the fledgling ERIC system indexed twenty-two references 
In Resources in Education (RIE) under the specific descriptor "cost- 
effectiveness". Now, more than ten years later, the annual average is 
nearing one hundred. In addition, nearly one hundred journal articles 

are indexed each year in Current Index to Journals In Education (CUE) 

« 

using the term cost-effectiveness. Reported applications of economic 
analysis procedures to instructional technology have shown a similar 
growth but on a more limited scale. What is there to be gleaned by 
educational technologists from such documents and articles on 
functional cost analysis, cost-effectiveness analysis, economic 
analysis, and program cost evaluation? 

In answer to this question, this ERIC Information Analysis Product 
is intended to present an overview of the current status of cost- 
I'f f ec t ivenc^ss cinalysis and related tlechniques as they should be and 
are being applied to the field of instructional technology. A selected 
and categorized annotatec. bibliography has been created to guide further 
^tudy. In addition, the references section has been expanded to assist 
iniiividuals seeking reports of applications in specific contexts as well 
as roviewR of concepts and techniques. 

Ci nations with FD numbers are available through th*^ ERIC Document 
Ktsroduct ion St^rvice in both microfiche and paper copy; if the ED number 
is followed by an asterisk, tha^" document is available only in micro- 
ticMio. Full information for ordering is included at the end of this 
pub 1 i ca t ion . 



INTRODUCTION 

Current conditions'' facing this country's educational systems make 
it imperative that we examine the cost-effectiveness of its instructional 
acitlvities and consider new ways of handling those activities. The 
educational system is in' a dilemma: On the one hand it is experiencing 
n financial crisis, and on the other it is subject to Increasing public 
criticism of both the ccst and the quality of its outcomes. To resolve 
this situation, educators must find new ways to maintain or improve the 
(juality of their services while more carefully controlling their costs. 

Instructional technology provides alternative approaches to the 
conduct of educational activities and cost-effectiveness analysis provides 
one means of evaluating the relative worth of those approaches in achieving 
dosired educational outcomes* Applications of instruc tioncil technology 
whirh are judged as having an appropriate relationship of costs to 
of f ec tiveness can be adopted by educators to help them maintain a proper 
balance between the cost and quality of their services. For this plan to 
work» however, instructional technologies must exist which represel)^ at 
leasi potential inprovements over conventional approaches and the methods 
of cost-effectiveness analysis must be good enough to determine the 
nature of those Improvements. 

Intended for practitioners with a basic understanding of instructional 
technology, this Information Analysis Product is designed to guide improve- 
ment in the design, conduct, and use of cost-effectiveness studies in this 
arc»a. Initial narrative sections examine the concepts and practice of 
eost-ef f ec t iveiu^ss evaluation and/or analysis as applied to instructional 
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technology, and provide an overview of the literature on this topic. 
Citations are included throughout this portion of the document so that 
more informed selections of follow-up literature can be made. 

For the user desirous of additional background infoimation, the 
appendix provides a review of the development of the practice of 
cost-effectiveness analysis of instructional technology programs and 
a brief review of that literature. Citations in this section are 
included in the list of references . 



COST EFFEgTIVENESS ANALYSIS 
Nature of Cost-Effectiveness Analysis 

•Cost-effectiveness analysis is one of a family of analysis activities 
known generally as systems analysis. Systems analysis is loosely related 
to the theory of general systems developed in the 1930 's by Bertalanffy 
(1968). It is based more directly upon work in the wartime field of 
operational analysis (Beilby, 1977a, p. 10; Quade, 1971, p. 10). In 
current usage, the concept of systems analysis encompasses several 
specific kinds of analysis activity, including cost-benefit analysis 
and cost-utility analysis in addition to cost-effectiveness analysis. 

Tlii;^ basic nature of systems analysis has been characterized as 
"an effort to apply structured rationality to problems of choice" 
(Dror, 1969, p. 6). Dror's reference to the "structured rationality" 
of the approach is illustrated in the following definition of systems 
analysis: 

Systems analysis may Le defined as inquiry to assist decision- 
makers in choosing preferred future courses of action by 
(1) systematically examining and reexamining the relevant 
objectives and the alternative policies for achieving them; 
and (2) comparing quantitatively where possible the economic 
costs, effectiveness (benefits), and risk of the alternatives. 
(Fisher, 1971 p. 6) 

The "problems of choice" Dror referred to provide the basis for 
distinguishing amons the various forms of analysis. Specifically, 
cost-effect i^yfnioss analysis is designed to assist a decision-maker faced 
with the problem of choosing among alternative approaches to a particular 
goal. Tht? relative desirability of the approaches is determined through 
a comparison of reKource requirements and outcomes of the different 
alternatives. Cost-benefit analysis is also concerned with the comparison 
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of the resource requirements and outcomes of alternatives, but this 

time the problem is that of choosing among different goals as well as 

among the means of achieving each. Finally, when tlfe problem of 

choosing between alternatives Involves more than the ^^termination of 

relative effectiveness and efficiency, ^nd becomes concerned with 

issues .relating to equity, social impact and the decision-maker's values, 

the most appropriate technique is cost-utility analysis (Tuschar , 497^) . 

These distinctions between the various types of analysis, however, 

are not usually this clear. All forms of analysis draw heavily upon 

a common set of theories and techniques from such areas as economics, 

engineering, and decision theory (Grosse, 1967, p. v; Quade, 1975, p. 22) 

The simplest way to understand any of these analysis types is to review 

the main characteristics they all have in common. 

/ 

Character istics of Anal y sis Activities 

Various authors (particularly Doughty, 1973; Fisher, 1971; Quade, 
19fS')b, 1975) have identified a number of characteristics of analysis 
activities that are common to all forms of analysis. In a sense these 
characteristics are the building blocks of the methods for cost-effective 
ness and other analysis efforts. Combining the observations of these 
anti*othor authors results in seven distinct characteristics of analysis 
ietivitles: pliase, purpose, function, level, variables, elements, and 
subsyxS terns. Each is described below. ^ 

Phases of an alysis activi t ies . Quade and Boucher (1968) and Fisher 
(1^^71) have defined Che basic process of analysis as proceeding through 
a series of iterations over five phases. 
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Formulation 

(The Conceptual Phase) 



Search - Including the 

Development of Hypotheses. 
(The Research Phase) 

Evaluation 

(The Analytic Phase) 



Interpretation 

(The Judgmental Phase,) 



Verification 

(The Testing Phase) 



Clarifying the objectives, defining the 
issues of concern, limiting the problem, 
searching out good criteria for choice. 

Looking for data and relationships, as 
w^l as alternative programs of action 
that have some chance of solving the 
problem. 

Building various models, using them to 
predict the consequences .that are likely 
to follow from each choice of alterna- 
tives, and then comparing the alterna- 
tive in terms of these consequences. 

Using the predictions obtained from the 
models and whatever other information 
or insight is relevant to compare the 
alternatives further, derive conclusions 
about them, and indicate a course of 
action. 

Testing the conclusions wherever possibl 
(Fisher, 1971, pp. 8-9). 



Of these five phases, Fisher notes that it is probably the fotmulation 
phase that is most critical to the success of the analysis. It is in the 
hancllinR of this first phase that the characteristics of purpose, function 
and level are of primary concern. 

Purpose. Analysis activities can be designed to serve three general 
purposes. As identified by Doughty (1973, pp. 16-18) the purposes are 
descriptive, predictive, and comparative. A descriptive analysis is 
conducted to describe the costs and outcomes of an ongoing program. 
Results from descriptive analyses coupled with other kinds of information 
(particularly that gained from modeling) can be used to conduct predictive 
ana lyses of the potential costs and outcomes of proposed programs. 
Finally, comj)ara tive analyses compare programs (alternatives) by employing 
either descriptive analysis for '*ex post facto" comparisons or predictive 
cinnlysls for a priori comparisons. Beyond these three characteristics 
purposes, there are a number of specific purposes which different types 



' of analysis are more or less suited to address. These specific purposes 
include "the allocation of resources among identified ob^eqtives . . . ^the 

w 

choice of alternative means to meet the given objectives... the assessment 
of the worth of different objectives... (and) the' systematic generation 
of alternatives which were not originally identified" (Doughty, 1973, 
p, 18). 

Junction . Analysis efforts are designed to aid decision-making. 
In particular, there are four important functions of decision-making 
which analyses can serve. These .functions have different purposes and 
otitcomes bat are closely interrelated (Doughty, 1973, p- 15). First, 
analysis may serve a control function when the decision-maker is con- 
cerned with managing and monitoring the flow of resources and level of 
output, Analysis activities may also support the function of planning 
for predicted changes in the resources and outcomes of activities. A 
third function is that of evaluation where the emphasis is on comparing 
thv desired versus the actual costs and outcpges of an activity. Finally, 
analyses may be conducted to serve a development function in order to 
assist in the generation, design, and implementation of new activities. 

Level. Another major characteristic of analysis activities is that 
thev should differ according to the level of decisions and information 
needs being served (Doughty, 1973; Gephart, 1971). An analysis conducted 
to assist top-level administrative decision-making will differ in depth, 
detail, and even tlie kind of information nrovided when compared to an 
analysis conducted at an intermediate or lower level of responsibility. 
Cenerally speak inp, the higher the level of responsibility and decision- 
makiny, the lower the amount of detail necessary and desirable in the 
reported data, 
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Variables * Glv^n the many variables involved In analysis (e.g., 
alternatives, time, cost, effectiveness) consideration Is typically 
given to fixing some set or subset of these variables in ortfer to 
simplify the comparisons and control for certain .types of errors. 
TKe two principal conceptual approaches are the fixed effectiveness 
and fixed budget (cost) approacKefi/t^^ p. 10). Under the 

fixed effectiveness approach, ct^e analysijs is structured so that the 
alternatives under comparison are all assumed to reach some specified 
level of effectiveness, and the emphasis is on determining which alterna- 
tive is least costly. The fixed budget approach compares s^ternatives 
on an equal cost basis to determine the alternative which is most likely 
to yield the highest effectiveness (or greatest benefit, utility). 
Multiple analyses are sometimes conducted on the 'Same "roblem in order 
to omploy both the fixed effectiveness and fixed budget approaches to 
•makr comparisons at several different levels of each. 

Tlie Offico of Education (DHEW) sponsored and disseminated a handbook 
for evaluation practitioners entitled The Resource Approach to the An alysis 
of ^Educational Project Co st (1978), which also recommends an approach to 
Vhv more typical situation where neither costs nor effectiveness are fixed. 
It suggests a '^^air-wise comparison" (p. 68) procedure similar to that 
employed by Doughty and Stakenas (1973) ar^d Lent (1976). In this situation 
the value judgments of the decision-maker (s) are specifically incorporated 
when deciding "whether or not the better outcome ^s worth the additional 
cost" (p. b9). 

E 1 emen t s o f ana ly sis. The analysis of any problem of choice either 
implicitly or explic-itly involves five elements: goals, alternatives, 
imparts, models, and decision rules (Quade, i575, p. 33). A goal (or 
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objective) Is the desired state or end-point which the declslon-tnaker 
wishes to realize. The alternatives are the possible means of attaining/" 
the goals. Each altarnative has various Impacts (cost, benefits, etc.) 
which accompany Its choice as a means of reaching the goal. The process 
of determining these xmpacts and comparing several alternatives Involves 
tho use of one or more models to predict the consequences of the alterna- 
tives In this 'Si tuation. Finally, the ultimate choice of one alternative 
owvr another depends on the use of decision rules which specify the basis 
upon which something in determined to be good or better than something 

Since analysis also involves judgm^ft it also involves certain 
elements of the process of judgment. These additional elements include 
criteria (aspects of something which are chosen as its potentially good- 
making or bad-making characteristics), standards (benchmarks for determining 
the adequacy or inadequacy of performance on a criterion), and Indicator s 
(mi'cins of determining something's performance on a criterion). Figure 1 
[)r(wides a simplified example of these elements in an analysis of a problem 
of i-hoice. rhr example was adapted from Quade (1975, p. 3A). 

Analys is subs ystems . For each of the main elements of analysis 
ident i f ied above (goals, alternatives, impacts, models, and decision 
rules), it is possible to identify separate analysis subsystems designed 
to identify or accompli sh the element. While most analyses do not treat 
t ach element in dc^pth, tliey all do employ one or more analysis subsystems 
to handle the determination of impacts. Two main subsysteins oi impact 
ana Ivs is are cost ana lys is and effectiveness analysis . 

Cost analysis Involves determining the negative impacts of something 
and, typically, valuing the extent of those impacts in dollars. The 



process of analysis is detailed and varies according to (slrcumstances, 
but generally involves the ex^raination of such factors as time, inheritance 
research and development requirements, and foregone opportunities. Beilby 
(1977b), Belmore (1973), Doughty and Beilby (197^>, Fisher (1971) and 



Seller (1969) provide thorough treatments of a variety of aspects of 
cost analysis. 

Effectiveness analysis is concerned with the positive impacts of 
something and particularly those positive impacts which are relevant to 
a judgment of the suitability of that something as a means to a given 
end. The analysis of effectiveness is at least as complex as cost analysis 
and perhaps more so since some components of cost may vary less from 
situation to situation than those of effectiveness. Determining the 
effectiveness of something involves defining the possible impacts of 
something, choosing indicators to measure those impacts, and establishing 
5^tandards for assigning value to performance on those measures. Specific 
methods of effectiveness analysis vary widely according to the nature 
and complexity of the decision being served by the analysis. Authors such 
as Carpenter and Rapp (1972), Scriven (1967), Seller (1969), Stake (1970), 
and Vadlmnapnnich (197(0 describe a range of issues and techniques that 
all fall under effectiveness analysis (particularly in an educational 
t on t ex I ) . 

A third analysis subsystem common to most complete analysis of choice 
is nn crrtal nty anal/sis, Quade (1975, p. 213) defines uncertainty as 
*'the unprodlrtabn i t ie.s in factors that affect the success of a course 
ot art ion." He gt^es on to describe the variety of ways in which different 
kinds of uncertaliuv can affect analysis and summarizes the points as 
t o 1 lows . 
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OBJECTIVE: 

ALTERNATIVES: 

MODEL: 

IMPACT: 
CRITERIA: 



To drive from home to work. 

The various routes from A to B. 

A map of the area shoving streets and 
freeways. 

Ease and cost of getting to work. 

(1) Number of miles in route. 

(2) Average speed limit of route. 

(3) Probability of delays due to coni»escion, 
traffic lights and stop signs. 



(For judgments related to criterion (3)): 
STANDARD: 



INDICATOR: 
DECISION RULE: 



The fewer the number of potential sto^ o, 
the lower th^ probability of delay. 

Number of Intersections indicated by the 
map for each route. 

The best route is the shortest one in , 
number of miles and travel time. (If 
one route is not the shortest in both 
mileage and time, in order to choose 
the best route, additional analysis is 
necessary to fix one side of the 
comparison, choose additional criteria, 
or weight the importance of the criteria.) 



Flv*urt» 1. Clements of a simplified analysis of choice 
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Therefore, uncertainty night be (a) conceptual ; What precisely 
is the problem? (b) factual : What are the relevant facts 
associated with the alternatives and the current situation? 
(c) predictive : What changes in the situation are likely to 
occur before any decision can take effect? And what are the 
likely consequences and reactions to the alternatives between which 
a choice must be made? (d) strategic : What counter actions 
may be expected to be taken by opposing interests? (e) e thical : 
What should the goals be and which of the potential outcomes 
would be preferable in the light of those goals? (p. 2,17) 

Some ot the specific methods for dealing with uncertainty include 

the use of Bayesian statistics to treat probabilities in judgments, a 

fortiori analysis to test conclusions under specific conditions of bias, 

and sensitivity and contingency analysis to test the consistency of 

results when changing criteria, standards, indicators, and/or other 

components of the analysis. Techniques for handling uncertainty are 

discussed in documents by the American Society for Engineering Education 

(1971); Edwards, Guttentag, and Snapper (1975); English (1968); Fisher 

(1971); and Swalm (1966). 



Character is tics of Cost-Effectiveness Analysis 

Earlier in this chapter, cost-effectiveness analysis was briefly 
doscrlbod in relation to other forms of analysis, i.e., systems, cost-benef 
nnd cost-utility. Having reviewed the characteristics that these forms 
of analysis hold in conmion, it is time to examine how they differ. 

Cost-effe ctiveness analysis distinguished from systems analysis . The 
Jifferenc(^ between systems analysis and cost-effectiveness analysis is not 
one of kind, but of degree or emphasis. Cost-effectiveness analysis is 
n particular instance of systems analysis where the predominant focus of 
the analysis Is on comparing the cost and effectiveness of alternative 
approaches to n goal. Narrowly defined, cost-effectiveness analysis is 



a component of systems analysis. More generally^ however, c6st-cf f ectlveijess 

analysis may expand to include all aspects of systems analysis in a 

particular situation. , 

Quade (1965a, 1975) has pointed out the differences between these 

broad and narrow conceptions of cost-effectiveness analysis. He describes 

the typical systems analysis as a very broad approach to the entire problem 

u. dor considerdt ion v^ith cos t-ef f ectivoness analysis as just one stage. 

Thus, characteristically, a systems analysis will involve a systematic 
investigation of the decisionmaker's objectives and of the relevant 
criteria; a comparison — quantitative where possible — of the costs, 
effectiveness and risks associated with the alternative policies^ 
or strategies for achieving each objective; and an attempt to 
formulate additional alternatives if those are found wanting. I 
regard the typical cost-effectiveness analysis as just one stage 
in this process. (1965a, p. 3) 

This Is ''cost-effectiveness analysis" used in its most narrow or specific 
sense. Elsewhere Quade notes that cost-effectiveness studies may be seen 
mure broadly as "any analytic study designed to assist a decision-maker 
Identify a preferred choice from among possible alternatives" (1965a, 
p. I), in which Cc-^se the only difference between systems and cost-effec- 
tiveness analysis is in the amount of emphasis placed on impact analysis- 

The distinction between systems analysis and cost-effectiveness 
analysis as narrowly or broadly defined is illustrated in the literature 
d«\s(:rihinR the eondurt of cost-effectiveness studies. For example, 
Kazanowski (1^68n» p. 115) describes cost-effectiveness analysis in the 
harrow sens*' as inrended "to identify the system whose capabilities meet 
t.ie mission req\iirements in the most advantt^geoiis manner." Thus it is 
a-isumed that xMv cost-effectiveness analyst works from stated goals (or 
mission reciui rement s) and th^es not begin by exploring the situation the 
w IV n sv<-itpmr an^ilysis would. Doughty and Stakenas (1^73, p- 175) view 
c )st-ef fee -iveness analysis more broadly as designed "to relate 
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quantitative and qualitative data in a systematic manner to facilitate 
making value judgments and decisions about programs." In keeping with 
this broader view, they believe that the first step of cost-effectiveness 
analysis is a descriptive study designed to investigate characteristics 
of the existing situation and thereby define the goalc and nature of the 
setting with which the analyst is to work. Doughty and Stakenas's first 
step in cost-effectiveness analysis thus corresponds directly with the 
first (or "formulation") phase of systems analysis. Aside from this 
difference as the beginning of analysis, both Kazanowski and Doughty 
and Stakenas's descriptions of the process of cost-effectiveness analysis 
follow Fisher's (1971) and Quade's (1975) view of the process of systems 
analysis. 

Cost-effectiveness analysis as distinguished from cost-benefit and 
cos t-utility analysis . One basic difference between cost-effectiveness 
and cost-benefit analysis rests in the nature of the decision situation 
being served. Following the discussions of Carman (1971, pp. 5-6) and 
Mayo, McAnany, and Klees (1973, p. 8), cost-benefit analysis is seen as 
appropriate for the larger problems facing an educational system; problems 
where the concern is more with what should be done than with how it 
should be done. Cost-effectiveness analysis is concerned with the issues 
of how. Of course not all authors or studies"' limit cost-effectiveness 
analysis to the study of means rather than ends, but the distinction 
tould become meaningful in the future. 

Cost-utility analysis differs from both cost-benefit and cost- 
effoctlveness analysis on the basis of what wil] be considered as part 
of the analysis; Tuscher (197A, p. 46) points out that neither cost- 
effectiveness nor cost-benefit analysis involves explicit consideration 
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of the decision-maker's judgments of program worth or value. As a result, 
ther« is sometimes a gap between the outcomes of analysis and the sub- 
sequent actions of the decision-maker. Cost-utility analysis tries to 
reduce this gap by employing the decision-maker's Intangible, subjective 
concerns in the analysis of optimal choice. Cost-utility analysis can 
be employed in addressing decisions that are either what or how in nature* 

These differences between the three forms of analysis have ramifications 
for the concepts and methods employed in the analysis which thereby create 
further distinctions between the approaches. For example, the comparisons 
conducted by cost-benefit analysis require that both positive and negative 
impacts of alternatives be estimated on the same scale. Thus cost-benefit 
analysis is an economic tool with an emphasis on estimating the dollar 
value of everything. The nature of cost-utility analysis, meanwhile, 
j)laces a corresponding emphasis on the techniques and concepts of decision 
tlioory. . Finally, cost-effectiveness analysis appears to be the most 
mothodo logical ly varied of the three approaches since the manner of 
effectiveness determination depends largely on the nature of the problem 
and setting. For typical uses of cost-effectiveness analysis with'lYT — -^^^ 
instructional settings, the techniques of educational program evaluation 
play a large role (Levin, 1^75). As has been previously pointed out, 
the^e three forms of analysis are alike to the extent that they are at 
least partially based in the techniques and concepts of economics and 
systems analysis. 

Tn summary, cost-effectiveness analysis is a distinct form of analysis, 
but one which sl)ar(»s certain things in common with other analysis activities. 
Am ^Hiade (19 7!^ p. 21) and others have noted, however, many of these 
distinctions disappear in practice. 
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LITERATURE REVIEW 
Implications from the Literature 

As stated earlier, a judgment that something is cost-effective or 
more cost-effective than something else involves a comparison of inputs 
to outcomes. Specifically, the comparison involves matching the resources 
expended on something <'usiially quantified in dollars) against the measured 
or estimated effectiveness of that something in achieving a particular 
goal or performance standard. Cost-ef f ec'tiveness analysis is viewed by 
most as the process by which one arrives at a judgment of cost-effectiveness, 
and in theory, the process is straightforward dnd logical (see Forbes, 
1974; Levin, 1975; Kazanowski, 1968a; or Alkin, 1970; for a general 
introduction to the approach). Problems aris'e, however, when this 
technique is applied within an educational context. As Levin (1971) 
has pointed out, we know so little about the nature and relationships 
between educational inputs and outputs that the possibilities for valid 
concluslohs about cost-effectiveness are severely limited. 

Difficulties facing cost-effectiveness analyses In education also 
apply to similar analyses of Instructional technology since its worth can 
only be judged upon its performance within an educational context. Most 
reported attempts to analyze the cost-effectiveness of instructional 
teclinology have therefore been of rather limited success or of limited 
.utility. 

Most studies reviewed here have avoided the complex issues surrounding 
the judgment of the effectiveness of instructional technology in achieving 
some educational outcome by falling victim to one or more of the following 
pitfalls: 

^0 



(1) They focus only on Inputs, 

(2) They focus only on costs (and not effectiveness), 

(3) They rely upon misleading thruput Indicators or proxies as 

measures of outcomes or productivity (such as cost per 
student credit hour), or 

(4) They rely upon single outcome focus as evidence of effectiveness* ^ 

More comprehensive considerations of pitfalls have been reported by Doughty 
(1979) and Kazanowski (1968b) but these four appear to be most pervasive. 
Brief comments on each are sufficient for the purposes of this report. 



(I) Studies which focus on Inputs can be most easily observed in* 
the many media comparison studies reported. These are usually 
investigations of changes in specific instructional media or 
techniques. Typically these studies are structured around a 
substitution of a new approach (e.g., instructional television) 
for an old approach (e.g., cortventional classroom lectures). ' 
Whatever was used as an outcome measure for the old approach 
(preferably something quantifiable, i.e., a final exam) is 
used as the outcome measure for the new approach. Little 
attempt is made to determine or compare the differing outcomes 
oT the two approaches. The relationships between inputs and 
outcomes are assumed to be linear and univariate. 

(!) Studies which focus only on costs. 

Some studies stress the requirement for cost-effectiveness 
judgments — but then proceed to fo.::us only on costs leaving 
the t»f f 1 i veness issues to the Judgments of the local decision 
makers (o.g., Jamison & Klees, 1975) or they cite the "no- 
«ipnitirnnt difference" research and claim that therefore the 



clioices can he made on the basis of cost alone (Caffarella, 



1^)7")^; Morris, 197A). At this point a sizeable literature 



has grown up around the cost-analysis of instructional 
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technology (IT) (e.g., Wagner, 1975; Wilkinson, 1973^ 
Chappell, 1970) and some work has been done to sumin^rlze 
this research and develop some general principles (e.g., 

' Caffarella, 1975b; General Learning Corp., 1968; Johnson 

, & Dietrich, 1971) for IT-focused practice. 

(3) Studies which rely upon misleading thruput indicators. 

The majority of cost analysis or unit cost studies (Witmer, 1972) 
conducted in higher education employ the standard unit of 
measure — the credit hour — as the measure of outcome or pro- 
ductivity. Rarely is this metric identified as a weak proxy 
or indicator of outcomes. Frequently this measure of student 
flow or thruput is combined with a cost figure to become a 
ratio of cost to or per student credit hour produced. Dis- 
cussions between legislators and academics about the quality 
and productivity of "higher education often include reference 
to a "high" cost-per-student-hour figure as an Indicator of 
quality or inefficiency when in fact it may be neither. The 
fact that, such ratios presently serve as primary data for 
decision-making in higher education should give cause for 
concern . 

(4) Studies that rely upon single outcome measures as evidence of 
effectiveness. Envision the many evaluation studies of in- 
structional methods or strategies that reported significant 
statistical and educational differences in achievement as 
measured by tlie X standardized test. Conveniently ignored 
was compelling evidence that those same high achievers be^an 
the study as eager learners and left as hostile students who 
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no longer care to pursue that subject or skill any further. 
The more enlightened program evaluation models, eonbined with 
functional cost analysis methods (Bellby, 1977a, 1977b) are 
helping considerably to eliminate this pitfall as well as- the 
Others previously mentioned. 
While much of this current literature is a valuable contribution to 
the state of the art, it falls short of providing evidence on the cost- 
effectiveness of instructional technology in two ways. 

(1) It fails to deal simultaneously with costs and effectiveness 

in comparing alternatives, and it 

(2) Fails to examine the C-E of IT when IT is defined as a 

systematic approach to instruction — rather than a 
collection of media techniques. 

Attempts to actually conduct cost-effectiveness evaluation of IT 
are continually confronted with the challenge of comparing the cost- 
effectiveness of alternatives when using multiple criteria of effectiveness. 
Attempts to do so usually rely on a matrix approach which displays the 
data on the var ious - criteria and leaves the final determinatioi) to the 

i 

decision maker. Attempts to infer any causal relationships between 
resources expended and results obtained are almost certain to be lost 
causes. Studies reported by Miller (1971), Doughty (1973), and Lent 
(1976) are examples of state-of-the-art methodology which do not attempt 
to make any causal links. 

Attempts^ to i^xamlne the cost-effectiveness of a systematic approach 
to instructional ttuhnology have Ln recent years come to mean more than 
the application ot assorted media in educational contexts. In the words 
of the Commission of Instructional Technology (1970) it is coming to be 
defined as "a systematic way of designing, carrying out, and evaluating 
the total process of learning and teaching in terms of specific objectives, 
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based on research in human learning and communication, and employing 
a combination of human and nonhumaili resources to bring about more 
effective instruction" (p. 19). When the IT field Is considered from 



the perspective of this definition, the problems of Conducting cost- 
effectiveness analysis multiply. Such questions arise as. What are you 



analyzing, the process or Its products? We now have the additional 
problem of determining the relationships betwt the application of 
this systematic process and changes In the Inputs... as well as rela- 
tionships between inputs and outputs. The conceptual complexity and 
number of unknowns in this arena is thus much greater.- 

There is almost no existing research in this area. Diamond et al. 
(1975) and Pearson (1972) have discussed some of the problems and possible 
models for work in this area. Belmore (1972) has reported his attempts 
at the cost analysis problems. Doughty and Stakenas (1972) have pro- 
vided one of the few documented attempts to conduct a comprehensive 
cost-effectiveness analysis of a systematic instructional project. 
Perhaps the considerable resources required to conduct such studies 
deters others from advancing research in this area, but If the power 
of technology Is in systems focused applications, then the assessment 
of those applications will require complex models of evaluation and 
cost-effectiveness analysis. 



Future Prospects 

The complexity of the problem facing both appliers of instructional 
technology and assessors of those apolicatlons must not discourage would-be 



researchers and analysts. There is a need for more substantive case 
studies of the cost-^t^f fee tiveness of Instructional technology and 
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Instructional development such that better operational heuristics 
can be developed. In addition, these studies can also contribute to 
the research and theory bases of boteh C-E and IT, as well as helping 
us to better understand what it means to be accountable. 

Other issues are evident in the literature, such as determining 
when it is advisable (cost-effective) to conduct a comprehensive C-E 
analysis. There is also the call for guidance in helping to determine 
when the notion of ''economy of scale" is an important factor in IT and 
C-E applications. More technical issues to be considered should include 
a shift from the emphasis on average costs to marginal or incremental 
costs — an important factor in C-E driven decisions. And perhaps the 
most important perspecMve for the study of instructional technology 
is the use of lifetime dollar costs as the basis for input comparisons 
rather than relying upon single cycle operational expenditures which 
oftentimes mask many of the actual costs of conventional instructional 
alternatives, as well as improperly distributing the start-up expenses 
of new innovative programs. 

The following references have thus been selected primarily on the 
basis of their contribution (or potential contribution) to advancing 
the practice of rost-ef fertiveness evaluation and analysis in Instructional 
technology. Al thouj^h many of the papers, reports and documents have been 
includt^d in the EKW. system, that circumstance had^little impact on our 
selections. However, in those instances where references of similar 
focus emerged, then the easy accessibility of ERIC documents served as 
n selection factor. 

It is Important here to acknowledge again the considerable and 
growing number of references in the ERIC system that embrace these two 
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areas of Interest, To assume that yOur collective experience and wisdom 

" *" * i ■. . • 

In the areas ensures the presence of all the best relevant documentdTin 

* ' ■» * '' 

this report is unwise. To' locate even better „^jjamples . of applications 

or more fundamentally sound guidance on C-E methods is everyone's goal. 

We trust that our colleagues in the ERIC netowVk will continue to 

assist us in that search — and vice-versa. 



o 
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ANNOTAT^ID BIBLIOGRAPHY ON COST-EFFECTIVENESS AND INSTRUCTIONAL TECHNOLOGY 



'It is useful to organize a bibliography in such a way that users 
can instantly identify those references that match their entry level 
knowledge and current interests. It was decided to organizfe the annotated 
cost-ef fectiteness bibliography under seven headings which collectively 
represent a general progression through a logical development of an 
economic analysis study from conceptual and theoretical concerns through 
to hard-edged outcomes. The seven headings are: 

(1) Concepts 

(2) General and Theoretical Approaches 

(3) Modeling Activities 

(4) Cost Analysis 

(5) Outcomes: Effectiveness and Benefits 

(6) Reviews and Critiques 

(7) Case Studies 

,Thls logical framework is attractive, but in many cases the actual 
references do not fall clearly into only one category. Several covered 
all the categories. To avoid the boring, and visually confusing 
reptjtltlon of refi?rences and op. cit ., the bibliography is presented in 
two sections: General References (Part I) and Case Studies (II). ^ 
.Nevertheless, a device which will enable the reader to identify Interest 



areas has been built into the bibliography: a number or series of code 
numbers occurs at the end of the annotation for each entry in-^the bibli- 
ography. These numbers — 1,2,3,4,5,6 — refer to the cor r-asponding categories 
);lv.»n above. Thus a rcierence followed by a 1, 3, and 4 indicates that 
"it addresses Concepts (1), Modeling Activities (2), and Cost Analysis (A). 
It is hoped that this simple organization will increase the 



L'l t or t Ivenuss and utility of tlie bibliography. 
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BIBLIOGRAPHY 



Part I - General References 



Carpenter, P. Cost-effectiveness as an aid to making decisions In education . 
Santa Monica, CA: Rand, 1970. (P-451^) (1,3) 

t 

Carpenter, M.B., Chester, L.G., Dordick, H.S. and Haggart, S.A. 

Analyzing the use of technology to upgrade education In a developing 
c ouatry , Santa Monica, CA: Rand, 1970 (RM-6179-RC) 

Two studies which demonstrate the application of systems analysis 
techniques to educational planning, and the potential usefulness 
of technology (especially ETV) in education in a developing 
country (Colombia). Four alternatives , for the future and potential 
upgrading of Colombia's primary and secondary education systems 
were analyzed. (1,3) 

Coombs, P.H. and Hallak, J. Managing educational costs * New York: Oxford 
Unlveisity Press, 1972. 

UNESCO's International Institute of Educational Planning funded a 
project to examine the use of educational cost analysis in various 
planning efforts. Twenty-seven case studies were eventually com- 
pleted and reported in three volumes as Educational cost analyses 
in action: Case studies for planners . The Coombs and Hallak 
book represents a synthesis of the conclusions and principles 
reached through these case studies. (1,A,5) 

Defense Economic ^Analysis Council, Handbook Committee. Economic analysis 
han dbook (2nd ed.). Washington, DC: U.S. Department cf Defense, 
unda^;ed . 

Provides a set of procedures to guide the uninitiated through the 
major phases of an economic analysis. Describes several costing 
methods and explains basic, relevant concepts. (1,3, A) 

Doughtv P.L. Cost effectiveness analysis tradeoffs and pitfalls for 
planning and evaluating instructional programs. Journal of 
Instructi onal Development , 2^, (4), 1979. 

The first article in a series on cost-effectiveness. Provides an 
overview of conceptual and practical criteria for judging and 
designing cost-effectiveness studies. (1,2) 

Doughty, P,L, and Beilby, A. Cost analysis and t^sacher evaluation: A 
comment oi i relevant relationships and a review of existing models . 
Albany, NY: The State Education Department, Division of Teacher 
Education and Cert if ication^ 1974. 

Describes thv system's context of cost analysis, reviews 19 cost 
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analysis studies and proposes a cost model' for use In teacher 
educa t Ion . (l,2,3,A»6) 

Doughty, P.L. and Stakenas, R.G. An analysis of costs and effectiveness 
of an Individualized subject offering. In CD. Sabine (Ed.), 
Ac coun t a b 1 1 1 ty ; Sy s terns pi ann ing in educ a t Ion . Homewood, IL: 
ETC, 1973. 

An investigation to develop and explicate methods for analyzing the 
costs, effectiveness and feasibility of Instructional alternatives. 
Results from a descriptive analysis were used in predictive com- 
parison which considered four future alternatives for course 
design: traditional small group instruction, conventional large 
group lecture-lab, individualized course with locally developed 
materials, and individualized course with commercially prepared 
materials. (3, A) 

« 

Doughty, P.L., Stern, H.W. and Thompson, C. Guidelines for cost-effectiveness 
analysis for navy tr a ining and educatio n. San Diego, CA: Navy 
Personnel Research and Development Center, 1976 (Special Report 
76TQ-12). 

Guidelines and procedures are presented to assist analysts and 
evaluators in determining when a cost-effectiveness study is 
feasible and appropriate, how best to conduct the analysis, what 
type of components to consider, and what kinds of answers to 
expect. Checklists, a Navy C-E case study and an annotated 
bibliography are also included*. (1,2, 3, A, 5,6) 

Kcksteln, 0. W ater resource development . Cambridge, MA: Harvard 
University Press, 1961. 

Classic work in the field of cost-effectiveness analysis. The 
first four chapters examine the theory of welfare economics as 
It applies to project evaluation. The remaining six chapters 
apply tho theory to various aspects of water resource develop- 
ment. (1,2) 

Kielden, J. and Pt^arson, R.K. Costing educational practice . London: 
Council for Kducational Technology, 1978. 

A handbook designed to describe how to do cost analyses. The 
case studies report cost analyses of a driver training program 
for the RAF, the reorganization of a university's laboratory course 
in biology, Mel f-lnstructlonal units in college-level science, and 
Sf?veral other innovative instructional projects. (1»A) 

Fisher, G.H. Co^t co nsiderations in system analysis . New York: 
AjTicrican Kisevier, Inc., 1971. 

Concepts and principles of systems analysis and cost analysis are 
defined in first three chapters while ^more explicit applications-, 
examples, and military oriented problems are discussed in the 
remaining six chapters. (1,2,3,4) 



2<) 



-27- 



Haller, E.J. Cost analysis for educational program evaluation,. In 
W.J. Pophan (Ed.), Evaluation in education . Berkeley, CA; 
McCutchan Publishing, 197A.- 

An introduction to cost analysis concepts such as life cycle, 
relevant costs and marginal costs. Contains reconmendations 
for measuring costs. (1,3,4) 

Jamison, p.T., Klees, S.J., and Wells, S.J. The costs of educational 
media: Guidelines for planning and evaluation . Beverly Hills, 
CaI sage, 1978. 

Summarizes various evaluation studies of the costs of instructional 
radio and television projects and offers a description of the 
methods and problems of educational cost analysis. Consideration 
is given to capital costs, student utilization over time, opportunity 
costs, shadow prices, the perspectives of decision-makers at 
different levels, and certain sources of error in cost estimation. 
(1,A,5) 

Layard, R. (Ed.) Cost benefit analysis . Baltimore, MD: Penguin Books, 
Ltd., 1972, 

Excellent anthology of important articles presenting theoretical 
basis of cost-benefit analysis including issues of benefit 
measurement, time preference, social opportunity cost, and 
treatment of risk. (2) 

y 

Lent, R.M. A model for applying cost-effectiveness analysis to decisions 
Involving the use of instructional technology. Journal of 
1"-S_tr_"c tlonal Development , 2» (H, 1979. s 

Part of a st'ries of articles on cost-effectiveness and cost analysis. 
This article provides a detailed model of the methods of cost- 
effectiveness analysis as used to inform decisions about instructional 
development efforts and other applications of educational technology. 
(1,2,3) 

Levin, H.M. Cost effectiveness analysis in evaluation research. In 
M. Guttentag (Ed.), Handbook of evaluation research , Vol. 2. 
Beverly Hills, CA: SAGE, 1975. ' 

A useful and clear treatment of cost-effectiveness. Introduces 
several economic concepts and discusses basic concerns relative 
to measuring effectiveness. (1,2,3,5) 

McKean, R.N. Effici ency in government through systems analysis . New 
York: John Wiley and Sons, Inc., 1958. 

{^ne of the classics In the area of benefit-cost analysis. Primary 
applied attention Is given to water resource decisions. (1,2,5) 
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NCHEMS Publications : Various works published by the Western Interstate 
Commission on Higher Education (WICHE), National Center for Higher 
Education Management (NCHEMS), present systematic procedures and 
practices for allocating costs and outcomes in higher education 
institutions. List of publications available from WICHE, Post 
Office Drawer P, Boulder, Colorado 80302. (3, A, 5) 

Page, E.B, Seeking a measure of general educational advancement. 
The Bentee. Journal of Educational Measurement , Spr 1972. 

An attempt to define a universal outcome measure. Raises issues 
concerned with values and measurement. (1|2) 

()uade, E.S. Analy s^ls for publ Lc decisions > New York: American 
Elsevier, 1975. 



Presents the concepts and procedures that comprise an analytical 
approach to decision-making. The framework for conducting analyses 
includes the following steps: (I) clarification of the problem; 
(2) identification of the objectives; (3) measurement of effective- 
ness; (4) determination of a criterion; (5) formulation of models; 
(6) data collection; (7) carrying out the comparison; (8) exam- 
ination of the analysis for sensitivities; (9) consideration of 
dt?f iciencies in the analysis; and (10) summarizing and, where 
appropriate, making recommendations. (1,2,3) 

Schramm, Big media little media; Tools and technologies for 

instruction , Beverly Hills, CA: SAGE, 1977. 

Chapter FoUr, "The Economic Evidence" (pp. 105-139), is particularly 
useful a general review of the costs of instructional media. 
Intended as a cnmpanion piece to the Jamison et al. (1978) text. 



Soilt^r, K. Intrn duction to systems cost-effectiveness . New York: 
WLloy- Intersc icnce, 1969. 

Though intoiuled for readers with a basic understanding of calculus 
and probability theory, issues and concepts are well described 
verbally. Spi>clfic sections focus on cost related issues, system 
cost models, an effectiveness model, and various comprehensive 
cost of f -^ctlveness analysis models. (1,2,3,4) 

rtakenas, R.C. and Kaufman, R. Costs and effectiveness of technologica l 
^l>pl Icat ion^ in education: A literature review , Tallahassee, FL: 
Florida Statt^ University, Instructional Systems Development Center, 

i^-;7. i-n It)') 922. 

An analy^iis i»r the findings from other studies concerned with 
technological applications in education. A good source for 
identifying other «=^tudies. (5,6) 

remkln, S. Making ^ense of benefit-cost analysis and cost-effectiveness 

analysis. Improv ing Human Performance: A Research (Quarterly , Sum 1974 
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A critical look at the language and assumptions of the two 
methods. Attempts to explain the methods In relatively 
untechnlcal terms. (1^2) 

Temkln, S. and McNamara, J.F. A comprehensive planning model for 

school districts; Decision rules and Implementation strategies. 
Journal of Educational Administration . Oct 1973. 

A planning model that Incorporates cost and effectiveness 
information is proposed. (Parts of the model were adapted by 
several school districts in the 1968-73 period.) (1,3,5) 

Wilkinson, G.L. Cost evaluation of instructional strategies. 
Audiovisual Communications Review , 21 , (1), 1973. 

One of the first articles to present a conceptually sound 
rationale for cost-effectiveness as a methodology in the 
evaluation of Instruction. Distinctions are made between 
various alternative cost analysis and cost evaluation 
techniques and the argument for cost-effectiveness analysis 
is presented. (1,2,3,4) 

Wilkinson, G.L. Economic evaluation in/or instructional development. 
In R.K. Bass and D.B. Lumsden (Eds.), Instructional development : 
The state of the art . Columbus, OH: Collegiate Publishing, 
1978. 

This chapter includes discussions about the concepts relevant 
to many types of analyses in instructional development contexts, 
as well as more detailed descriptions of cost-effectiveness 
and cost-benefit evaluation. (1,4) 
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r^art II - Case Studies 

7 



Public School Case Studies ^ 

Belmore, W.E. The application of a cost analysis methodology to the 
design phase" of instructional development (Doctoral dissertation, 
Indiana University, 1973)., Dissertation Abstracts International, 
1974, 33, 5996-A. (University Microfilms No. 73-10,-756} 

The development and testing of a functional cost analysis procedure. 
Cost data were collected only for the design phase. 

Ernst and Ernst. A model for the determination of the costs of special 
education as compared with that for general education . Chicago, IL: 
Ernst~and Ernst, Feb 1974. 

A model for determining special education costs. Includes some 
empirical work and employs an accounting perspective. (1,3,4,5) 

Callltls, M.M. The costs of information retrieval television; A case 
study in the cost-effectiveness of educational media . (Occasional 
Paper No. 12). Toronto: Ontario Institute for Studies in 
Education, 1972. 

A detailed study of the cost and cost-effectiveness of an 
information retrieval television (IRTV) system when compared 
to other systems for delivering a similar service. The study 
provides both descriptive and predictive comparative analyses 
of system cost per unit of system performance (defined as 
system use ) . ^ 

Michigan Department of Education. Research, Evali^tion and Assessment 
Services. Report of the 1974-75 Michigan cost-effectiveness study . 
Washington, DC: Capital Publications, Inc., Educational Resources 
Division, 1976. .1 77p. ED 134 634. 

A study to identify factors that', contribute to more effective 
and efficient special educational delivery systems. (3,5,7) 

Rorro, CM. A cost study analysis of school district implementation of 
IS V. in N ew l^rF\ey , 1978. (Unpublished doctoral dissertation at 
Fairlcigh-Oickinson) . 

A rttudy of (^osts incurred by A2 school districts in New Jersey as 
they impLomented the goal setting pha^e of the state's T&E approach 
to school impro>;ement . (A, 7) 



Post Socondarj/^ Kducatio n Case SLudles 

Balt^y, J.D. Cost effectiveness of three methods of remedial instruction 
in mastery lenrninK and the relationship between aptitude and 
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achievement. (Doctoral dlBsertation, University of Southern 
California, 1972). Dissertation Abstracts International . 1973, 
33, 3A75-A. (University Microfilms No. 73-716) 

A study comparing the relative cost effectiveness of three methods 
of remedial instruction in mathematics and tutorial assistance by 
the course instructor. (1) 

Hi 

Bellby, A. A functional cost analysis process applied to selected 

competency-based teacher education programs in New York State. 
(Doctoral dissertation, Syracuse University, 1977). Dissertation 

Abstracts International . 1978, 39, 554-A. (University Microfilms 
No. 7811633) 

The development and test of a functional cost analysis procedure. 

Bellby, A.E. Costing instructional programs . Ithaca, NY: Cornell 
University, Cornell Institute for Occupational Education, 1978. 
103p. ED 165 333.* 

Introduces functional cost analysis ^nd life-cycle costing 
concepts. Two detailed case studies of two-year college 
Instruction programs are presented. (1,3,4,5) 

Doughty, P.L. Effectiveness, cost, and feasibility analysis of a 
course in college level geology. (Doctoral dissertation, 
Florida State University, 1972). Dissertation Abstracts 
International , 1973, 33, 5A67-A. (University Microfilms 
No. 73-10, 325) 

Compares four Instructional approaches (including lecture-lab 
and individualized audio-tutorial) in a university setting as 
a means of validating a cost-effectiveness model. (1,2,3,4,5) 

Enos, D.F. A cost-effectiveness analysis of c6mpetency and non-competency 
based teacher education at San Diego State University (Doctoral 
dissertation. University of Texas at Austin, 1976). Dissertation 
Abstracts Internationa l, 1976, 21. 2793-A. (University Microfilms 
No. 76-26, 625). 

A comparison of the cost and effectiveness of two means of 
preparing prospective elementary school teachers at San Diego 
State University. 

Lent, R.M. Planning for future University College programs. A cost- 
effectiveness evaluation of alternative instructional delivery 
systems . Syracuse, NY: University College of Syracuse University, 
1976. 120p. ED 142 195. 

,JLent, R.M. Program planning and the cost-effectiveness analysis of 

instructional technologies: A case study In planning continuing 
education services . Paper presented at the annual meeting of the 
American Educational Research Association, New York City, 1977. 
49p. ED 145 801. 
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These papers explore the strengths and weaknesses of "^alternative 
systems for delivering continuing education courses to students in 
central New York* The evaluation was conducted within a framework 
of six costs and effectiveness criteria with the performance of 
the alternatives compared on a Cost-Effectiveness Comparison 
Matrix. 



Military Case Studies 

Bellby, A.E., Locatis, C, Marvin, M,, and Wilbur, F, An economic u 
analysis of TRADOC's basic non'-commissloned officers coutse for 
the combat arms program . Fort Monroe i VA: U.S. Army Training 
and Doctrine Command, 1978. 

The primary objective of this study was to provide decision 
makers with data to enable them to select the most cost- 
effective combination of non-commissioned officer training 
site locations to meet Army cost and delfense strategy goals. 
A secondary objective of the study was to identify problems 
which may have existed with the design, development and 
delivery of officers' training. Specific recommendations 
were made regarding optimal class size, establishment of 
training locations, development of indicators of course 
effectiveness, adoption of alternate instructional systems 
and materials, and other issues relating to maximizing the 
efficiency of instruction. (1,4) 

T€>mkln, S., Connolly, J., Marvin, M. , Valdes, A., and Caviness, J. 

An economic assessment of Army training alternatives . Arlington, 
VA: Army Research. Institute, 1975. 

A studv of the costs of the use of an audiovisual training approach 
compared with those of conventional instruction in the Array. 
(3,4,5,7) 



Intt^rnational Cas e Studies 

Coombs ♦ P.H. and Hallak, J. Educational cost analysis in action: Case 
studies for planners - 1, II, III . Paris: UNESCO International 
'institute for Educational Planning, 1972b. Vol. T, 356p., ED 082 337*; 
Vol. 2, 27np., ED 082 338*; Vol 3, 343p. , ED 082 339.* 

A three volume series of cost analysis case studies conducted of 
educational projects throughout the world. Companion pieces to the 
synthesis report Managing educational costs (Coombs & Hallak, 1972a), 
(7) 

Mayo, J.K., McAnany, E.C., and Klees, S.J. The Mexican Telesecundaria ; 
A cost-off cct tve ness analysis . Stanford, CA: Stanford University 
Institute for Communication Research, 1973, 173p. ED 076 059. 
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Maycf, J.K., McAnany, E.G., and Kle68, S.J. The Mexican Telesecundarla : 
A coat-effectiveness analysis. Instructional Scienc e. 1975, A, 
193-236. " ~ 

An evaluation of a six-year old project designed to provide 
secondary education to rural students in Mexico through the use 
of television. The evaluation compared this project, the 
Telesecundarla system, against the traditional secondary 
education system, Ensenanza Directa, on a number of cost and 
effectiveness variables. Telesecundarla was found to be com- 
oaratively inexpensive, effective, and efficient in fulfilling 
its functions. 
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Instructional Technology and Cost-Effectlvencii Analysis 



Characteristics of Instructional Technology 



The Association for Educational Coimnunlcations and Technology 
(1972, p. 36) has broadly defined instructional technology as the field 
devoted to the "facilitation of human learning through the systematic 



range of learning resources and through the management of these processes." 

Some specific applications of instructional technology Include computer 

assisted instruction, programmed instruction. Instructional television 

and instructional development. 

Many advantages have been claimed for the use of instructional 

technology. Those studying its potential future Impact on education 

have stressed its potential ability to improve the productivity as 

well Qs the quality of educational' programs. 

With the demand for education outstripping education's Income, 
more effective and efficient learning is vital. Instructional 
technology has shown its ability to speed up the rate of 
learning. It can help the teacher make better use of his time. 
It can reduce the teacher's routine job of information trans- 
mission. Thus the teacher would be able to spend more time 
on teaching . . . (Commission on Instructional Technology, 1970, 
pp. 30-31) 

Although short-run coats for the development and introduction of 
new instructional technology are expected to be very great, they 
will ultimately yield dividends. Much of the expanding technology 
has the potential economic effect of spreading the benefit of 
Investment in a single unit of instruction among very large 
numbers of students. It therefore has the ability to increase 
the productivity of higher education. The earlier this in- 
creased productivity will be realized... (Carnegie Commission 
on Higher Education, 1972, pp. 45-46) 

To test such claims for instructional technology, various writers 
have called for evaluations of Its cost-effectiveness. 



identification, development, organization and utilization of a full 
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We strongly recoBonend that broad-baaed studies be made of the costs 
and benefits that can be expected if the various technologies in- 
volving audiovisual equipnent, television, computers, and other 
devices are applied ,to instruction in the schoola on a wide scale. 
Such studies should take into account the benefits that nay be 
obtained through increasing the effectiveness of the learning 
procers at the same tine that they weigh the effects of the new 
resources in terms of the organization of instruction, teacher 
pay schedules, productivity, probable use by teachers, and other 
vital matters. (ConmitteA for Economic Development, 1968, p. 19) 

Ue need to experiment with Instructional management arrangements 
that permit mediated instruction to pay for itself. Cost-effective 
ness information is a first step in buttressing arguments for such 
arrangements. Much more research, time, and effort are needed in 
this area. (Helnich, 1968, p. 222) 

If the full potential of technology in education is to be realized'v 
well-prepared cost-benefit and cost-eff ectiveness , analyses are 
necessary. (Grayson, 1972, p. 1216) 



Cost-Effectiveness Defined 

Determining the cost-ef f iectiveness of something (such as an application 
of Instructional technology) is a unique kind of evaluation activity. 
U'hlle definitions differ, cost-ef fSctiveness can generally be defined as 
the riilationship between something's inputs (costs) and outcomes (effective- 

i 

ness) relative to the particular goal being served. Cost-effectiveness 

is a distinct characteristic or attribute of something that can bo Judged 

and compared a^gainst the similar attribute of another thing (Carpenter, 

1970, p, 17). One alternative is judged as more cost-effective than 

« 

another if, for example, it is more effective in reaching the goal for 
a given level of cost or if it reaches a fixed level of effectiveness 
for the lesser cost (Quade, 1975, p. 25). 

Quade (1971, p, 2) groups cost-effectiveness analysis with cost- 
benefit analysis, policy analysis, operations research and other management 
sciences as sharing the common purpose of aiding decision making. Cost- 
effectiveness analysis Is dls tine tive , in that it is designed to compare 
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alternative approaches to a given goal. The cost of the alternative 

under consideration can generally be represented in monetary units. 

Effectiveness Is usually not « estimated in dollars, but rather is ij^easured 

on a scale chosen to reflect the nature of the particular goal (I.e., 

achievement test scores) • Thus, while cost-effectiveness analysis Is a 

parftlcularly flexible technique. It is only suitable for choosing among 

competing approaches to the same goal* When choices have to be made 

between competing goals as^^rell as alternative activities, effectiveness 

must \)e measured in the same units as costs in order to make a meaningful 

comparison. Under these circumstances, the more specialized economic 

tool of cost-benefit analysis (comparing costs and benefits on- identical 

scales of estimated monetary value) IS more approprl.ate (Quade, 1971, 

PP- 2-3). ' ^ 

Cost-of fectlveness analysis is, quite frankly, a technique for 
comparing programs, and may be used: 

-to help assess the relative worth of several innovative programs 
Kith the same educational outcome (such as improvement In 
reading achievement) ; 

-to determine whether a single program is becoming more or less 
effective >as time passes so that steps may be taken to improve 
it. If necessary; 

-to help assess the relative worth of the same program for 
different student populations (such as those with differing 
socioeconomic backgrounds) or in different school settings. 

The ability of cost-effectiveness analysis to assist in these kinds of 

decisions could impact upon the future use of instructional technology as 

an innovative or alternative approach to the problems facing education. 



Development of Cost-Effectiveness Analysis in Education and Iiitstructional 
Technology 

Concepts and methods of cost-ef fectlveness#analysis have been derived 
from economic theory, practical engineering and systems or operational 
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analysis (Quade, 1971, p. 8). Cost-effectlveneas analysis has nuch In 
common with the older technique of cost-benefit analysis but, according 
to Quad.e (1971, p. 11), it eyolved quite separately as a result of 
wartime work (1940's) In operational analysis. V^lthln a few years, . 
both techniques had been Introduced to many analysts. 

The methodology of systems analysis and% specifically, cost-* 
effectiveness analysis developed slowly through the 1950's and early 
I960' 9 with applications to water resources projects (McKean, 1958) 
and military planning (ilitch & McKean, 1963). Support for analysis 
activities increased markedly when Charles Hitch became Assistant 
Secretary of Defensd under Robert S. McNamara and a group of analysts 
and economists 'were bro\ight to the Department of Defense. Within a 

at 

few years analysis concepts began to spread throughout the federal 

Rovernment. In August L965 President Johnson made the following 

announcement during a news conference: 

This morning I have just concluded a breakfast meeting with 
the Cabinet and with the heads of Federal agencies and I am 
asking each of them to immediately begin to introduce a very 
new and very revolutionary system of planning and programming 
and budgeting throughout the vast Federal Government, so that 
through the tools of modern management the full promise of a 
finer life can be brought to every American at the lowest 
possible cost. 

Under this new system each Cabinet and agency head will 
set up a very special staff of experts who, using the most 
modern methods of program analysis will define the goals of 
their department for the coming year. And once these goals 
are established this system will permit us to find the most 
effective and the least costly alternative to achieving 
Apierican goals. (The New York Times, August 26, 1965, p. lA) 

The application of cost-effectiveness analysis to education basically 

dates to President Johnson^s directive and from that time oh the adoption 

c 

of the technique is readily apparent. For example, from 1956 to 1967 

Office of Eduiia tion Research Reports listed four publications related 
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to' analysis activities. In 1968, the new Educational Resourcea 
Information Center (ERIC) listed 22 references under the specific 

descriptor "cost-effectiveness" and by the 1970' s the number of 

« 

citations averaged 90-100 each year. The spread of cost-effectiveness 
analysis to the specific area of instructional technology followed a 
similar pattern. The first report of cost analysis of instructional 
media appeared In 1965 (Barson & Jones) and the term "cost-effectiveness" 
was first used in the professional Journal Educational Technology a few 
years later <Caldwell, 1968). 

As cost-effectiveness analyses began being conducted in education, 
however, a number of writers (e.g.»Cogan, 1971; Grayson, 1972; James, 
1969; Lovell, 1971) begin noting particular difficulties in adapting 
the methods of cost-^ef fectiveness analysis to educational settings. 
j;enerai.ly» these difficulties were seen to reflect the rather messy 
and ill-defined nature of educational decision situations as compared 
to the military and engineering contexts within which the technique was 
originally developed (Quade, 1975, p, 8). Mushkin and Cleveland (1968, 
p. 90) Identified four factors which make analyses in educational setting 
particularly difficult: 

(1) The long gestation period of educational outputs and the 
length of the necessarily sequential learning processes. 

(2) Our limited knowledge of the learning process ^which might 
hamper attempts to attribute a particular result to the 
actual activity which produced it. 

(3) The multiplicity of objectives in education which complicates 
^ the task of assigning a particular activity to the final 

educational purpose which it serves. 

(A) The difficulty of factoring out the effects of non-school 
experiences ah the V^ocess and product of learning. 



54 0 



In a detailed crltlquei Levin (1971) outlined the problems facing 
the cost-'ef fectiveness analysis of Instructional technology and concluded 
that "most of the req^uirements for sound cost-effectiveness analysis 
cannot be satisfied given our present knowledge of the educational 
process" (p. 1000)* This conclusion was .arrived at through his inter- 
pretation of education as a production process where educational goals 
define the outputs of the process; students, teacheifs, and other resources 
define the inputs to the process; and the various inistruc tional strategies 
and other techniques for combining these Inputs define the means of 
pr oducing the various outcomes. He argued that although current knowledge 
and theory in any of these areas is too limited to support the conduct 
of cost-effectiveness studies, the effort must continue provided that we 
"recognize the formidable heiglit of the barriers placed in the path of 
cost-effectiveness analyses in education rather than pretending that 
our hurdling ability is sufficient over any course" (p. lOOA), 

Cost-effectiveness analysis has now been part of the lexicons of 
education and instructional technology for over ten years. During this 
period there have been marked advances in the theory and methodology of 
the closely related fields of educational evaluation and cost analysis 
(e.K,, Belmore, 1972; Coombs & Hallak, 1972a; Glass, 1969; Haller, 1974; 
Scrlven, 1967; Stuff lebeam, 1971; Witmer, 1972). In addition, a con- 
siderable literature has grown up around the methodology of educational 
COS t-ef f ec t Ivenesfi analysis with some particularly notable contributions 
by Alkln (1969), Doughty and Stakenas (1973), Levin (1975), and Temkin 
(1^)69). Uiffortunately , all this activity seems to have had a limited 
effect upon the actual conduct of cost-effectiveness studies in education 
and specifically instructional technology, at least as such studies are 
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reflected in the literature. 

Previous Reviews 

There have been four recent attempts to summarize or critique 
available studies on the cost-effectiveness of instructional technology. 
The most-comprehensive survey was conducted by Caffarella (1973, 1975a, 
1975b), who identified over 400 references i.i this general area. After 
reviewing approximately 300 of these references, Caffarella found only 
32 that reported either empirical studies or included findings supported 
by quantitative data. Much of the literature consisted of expositions 
regarding the requirement and methodology for cost-effectiveness studies 
He concluded that, "The cprrent quantity and quality of research on the 
cost-effectiveness of instructional technology la low... It is evident 
that even though there has been a proliferation of writings in the 
cost-effectiveness of instructional technology there has been very littl 
research on the subject" (1975a, p. 15). Carnoy (1976) and Carnoy and 
Levin (1975) have conducted more detailed examinations of a smaller, 
selected group of cost-effectiveness studies. In his 1976 article, 
Carnoy examined both the analytical and empirical bases upon which 
instructional television is being justified as a solution to the problem 
of providing 'Equality" education to people in developing countries, 
l^ile this study did not focus strictly on cost-effectiveness investi- 
gations, Carnoy did describe a number of flaws in past applications of 
the technique in television studies and suggested better ways to conduct 
such studies in the future. 

The Carnoy and Levin (1975) article reviewed six investigations on 
the costs and effectiveness of educational media (radio, television, 
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computer assisted instruction* and video*taped instructional systens). 

Their review mainly explored the Impact of two kinds of bia& ("benefit 
of the doubt" and "ignoring other effects") on the conduct and con- 
clusions of thiese six studies. Finally » Rogers (1976) reported on 
problems he found In three studies on the cost and/or cost-effectiveness 
of instructional technology. This analysis focused exclusively, on 
methodological probletas In the costing of instructional technology. 

These reviews seem to suggest two things about the state of cost- 
effectiveness analysis of instructional technology. First, it appears 
that relatively few studies have actually been conducted (assuming that 
most formal studies have found some, publication outlet). In fact, this 
situation may be even more severe than Caffarella's findings suggest. 
A careful review and updating of his 1973 study suggests that Caffarella 
used the most liberal interpretation of what constituted a cost-effective 
ness study. If only those studies are considered which are specifically 
recognizable as attempts to Investigate cost-effectiveness, then the 
total number of empirical investigations of Instructional technology 
reported to date Is approximately 15 (depending on definitions of 
"empirical" and ''instructional technology"). The second conclusion 
that can be drawn about the reported studies is that they are character- 
ized by a variety of shortcomings. For example, both Caffarella (1975b) 
and Carney (I97ft) have pointed out the general failure of most studies 
to deal with the whole effectiveness side of the comparison. 

One possible (explanation for the poor state of the empirical 
literature is sugv,^Rted by the nature of the methodological discussions 
Intended to explain and support the conduct of such studies. From even 
a cursory review of this literature, it becomes apparent th.^t there is 
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confusion and disagreement over some the moat fundamental concepts 
of educational cost-effectiveness analysis. For example, there are 
conflicting recommendations as to the basic structure of the analysis. 
Some authors (e.g., Carpenter & Haggart, 1972) argue that, In educational 
settings, analyses should be conducted on a fixed cost basis which compares 
the effectiveness of alternatives at a given level of resource use. 
Taking another point of view, other analysts favor fixed effectiveness 
comparisons which analyze costs at a set level of effectiveness (e.g., 
Grayson, 1972). Still other analysts advocate the use of both fixed 
cost and fixed effectiveness comparisons (e.g., Kazanowski, 1968a), 
or they argue for the use of a variable approach that fixes neither side 
of the comparison. 

Some of this confusion and disagreement is less striking, perhaps, 
when one realizes that there is no standard definition for what con- 
55titutes a ^'cost-effectiveness analysis." The definitions employed 
by different analysts often vary in subtle but important ways. For 
example, in the quotations given below the definition used by the 
National Center for Educational Statistics emphasizes resource analysis 
while thfe definition offered by Diamond et al. stresses the determina- 
tion of effectiveness, and Meeth*s definition stresses the interaction 
between both sides of the analysis. 

Cost-effectiveness - Analyses designed to measure the extent to 
which resources allocated to a specific objective under each of 
several alternatives actually contribute to accompliahing that 
objective, so that different ways of gaining the objective may 
be compared. (National Center for Educational Statistics, 1975, 
p. 21/4) 

Cost-effectiveness analysis Is a process which attempts to 
determine the most effective and efficient way of reaching 
the goals that have been identified* The key problem, there- 
fore, Is to identify and measure those elements which together 
describe the effectiveness and efficiency that is required. 
(Diamond, Elckmann, Kelly, Holloway, Vickery, and Pascarella, 
1975, p. 113) 
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It la the act of comparing the relationship between input and 
output between the resources and their related dollar costs 
and the achievement of desired goals, competencies, of other 
outcomes... Cost-effectiveness analysis is a qualitative 
judgment made about the relationship of cost to outcomes. 
(Meeth, 1975, p. 124) 

With this kind of variation in the methodological literature it 

seems hardly surprising that the cost-effectiveness analysis of 

instructional technology has proceeded at a relatively slow pace In 

terms of the conduct of actual studies and the further development of 

new methods. This, however, Is not the first time that the confused 

state of the methods for analysis has been noted and cited as an 

important problem that needs to be resolved. In what is becoming a 

classic article in the field, Kazanowski (1968a) argues the necessity 

of a standardized approach to cost-effectiveness evaluations. 

The lack of uniformity that ts apparent* in almost any sample of 
cost-effectiveness evaluations has resulted in a questioning of 
the merits of cost-effectiveness as an aid to decision-making. 
Although it is recognized that numerous approaches to cost- 
effectiveness evaluations may generate equally valid results, 
the plethora of different approaches, when combined with 
questionable evaluations, unavoidably generates a skepticism 
of the merits of cost-effectiveness analysis in the minds of 
the recipients of the evaluations. Thus what could be a 
valuable decision aid becomes a questionable technique 
viewed with considerable skepticism. (p. 113) 

Shortly thereafter, Temkln (1970) reached similar conclusibns specifically 

in rep.fjrd to cost-^ef fectiveness studies in education: 

In the past few years countless authors have suggested, in one 
way or another, that cost-effectiveness methods be used as a 
basis for decision-making in education. As one sorts and 
analyzes these proposals and prescriptions two important 
Inadequacies become evident: 

(I) Firm theorotical bases for these studies are lacking. 
Theory, in the domain of decision-making, should pro- 
vide not only a basis for description and explanation 
but explicit statements of assumptions underlying the 
proposed rationale and methodology. 
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(2) Little help is dffered to the individual who wishes to 
select from the various economic based approaches an 
appropriate method to apply to a practical problem, 
(p. 1) 

In spite of the efforts of both df these authors to remedy the 
situation, it continues today. In 1976 Vadhanapanlch reviewed the 
state of the methods for analyzing instructional technology's cost- 
effectiveness and concluded that, "The procedure of cost-effectiveness 
is not well understood; because of the lack of methodological and 
conceptual guidelines for analysis of instructional technology, the 
cost-effectiveness approach has not yet been widely implemetjted'* 
(p. 2). 

Thus, after more than ten years of activity directed towards the 
cost-effectiveness analysis of instructional technology, useful studies 
in the area are relatively few in number and uncertain in quality. The 
concepts and procedures involved in conducting the studies remain un- 
certain in many instances. In short, it is not clear what progress has 
been made in adapting the methods of cost-effectiveness analysis to the 
particular circumstances surrounding evaluations of instructional 
technology. Further work is required in developing, clarifying, and 
standardizing the practice of cost-effectiveness analysis if it is to 
be truly useful and more widely employed in decision-making. 

As other analysts have noted these problems they have mounted their 
own efforts to* resolve them. Kazanowski (1968a), Temkin (1970), and 
Seller (1969) have all made considerable contributions to the general 
methodology of cost-effectiveness analysis. Studies by Pearson (1972) 
and Vadhanapanlch (1976) have aimed at unifying and clarifying methods 
specifically devoted to the analysis of Instructional technology. In 
spite of the work of these and other analysts, however, the problems 

so 
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persist. Further; work and new approaches to methodological study are 
needed In this area. 

MQthodological studies of cost-effectiveness analysis have typically 
begun with the development of a hypothetical model for analysis based on 
various bodies of existing theory. This model Is then- variously subjected 
to demonstration, review or validation under real or contrived circum- 
stances. The usual Intent is to document the worth and generalizability 
of the proposed model. In a few instances' (e.g. , Doughty, 1973) another 
approach has been used which begins with a particular problem for analysis, 
develops or adapts methods and techniques for analyzing that problem, 
reports the analysis, and concludes by discussing the wortih of the 
study's process and findings. So far neither approach has produced 
generalizable procedures or results that are robust enough to survive 
the reality of cost-effectiveness analysis In educational settings. 
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